The present invention relates to an image sensing control method and apparatus for remote-controlling an image sensing device such as a still camera, video camera, and the like, an image transmission control method, apparatus, and system for transmitting the sensed image, and a computer readable storage medium that stores a program for implementing the method.
A camera control system in which a plurality of cameras each having pan and tilt functions and a zoom function are set at the respective positions, and an operation device at the central position or an arbitrary position is connected to these cameras via a dedicated communication line to remote-control an arbitrary camera, is known. In such system, control signals (cables or signal lines) for transmitting camera control/status signals are required in addition to cables for transmitting video/audio signals. Also, a method of modulating these camera control/status signals and superposing them on video/audio signals is known. Theoretically speaking, however, such method is equivalent to a method of assigning separate communication channels to individual signals.
In such camera control system (a system that remote-controls cameras using the dedicated communication line), an apparatus for remote-controlling a camera is fixed in position, and becomes a bottleneck against easy system expansion.
For this reason, it is preferable to remote-control cameras on a network built by computers and the like. This is because both an apparatus for controlling a camera and an apparatus for remote-controlling a camera operate as members of a computer network (to be simply referred to as a network hereinafter), and users can freely remote-control these apparatuses from terminals (e.g., personal computers or the like) connected to the network.
On the other hand, a camera conference system that transmits audio signals onto the network in addition to images sensed by the cameras is receiving a lot of attention.
FIG. 1 schematically shows a general image communication system including the above-mentioned camera conference system. FIG. 2 is a schematic block diagram showing the arrangement of the image communication system shown in FIG. 1. A communication terminal 100 serves as an image server for controlling, e.g., a video camera and transmitting the image sensed by the camera comprises a computer main body 101, a keyboard 102, a mouse 103, a display 104, and a video camera 105, as shown in FIG. 1. A communication terminal 200 serving as an image client which receives image data from the communication terminal 100 and displays the received image comprises a computer main body 201, a keyboard 202, a mouse 203, and a display 204, as shown in FIG. 1. The communication terminals 100 and 200 (client) are connected via a network 300 to be able to communicate with each other.
As shown in FIG. 2, the image sensed by the video camera 105 of the communication terminal 100 is converted into digital data by an image input unit 110. The digital image data for one frame is packetized by a network transmitter/receiver 111, and the obtained packet is output onto the network 300. The packet output onto the network 300 is received by a network transmitter/receiver 211 of the client (communication terminal 200). Upon reception of the packet, the client converts the received packet into data, and stores the data in an image buffer 210. The client reads out the image data from the image buffer 210 at a predetermined timing, and displays the image on the display 204.
By repeating the above operation, moving images captured by the communication terminal 105 via the video camera 105 can be displayed on the client side. The image data is stored in the image buffer 210 for the following reason.
A packet on the network has fluctuation (jitter) with respect to time for various reasons. FIG. 3 shows this state. As shown in FIG. 3, even when the transmitter side of image data transmits the image data for one frame at a predetermined interval T, the receiver side does not always receive the image data at the predetermined interval T due to the influence of the jitter. For this reason, the reception interval may broaden, as indicated by A in FIG. 3, or may narrow, as indicated by B in FIG. 3.
Hence, if the client displays the received data immediately without going through the image buffer 210, the motion of images displayed on the display 204 may be too slow (the portion A) or too fast (the portion B). In order to prevent such irregular motion, image signals for a time that can absorb the jitter are normally buffered in the image buffer 210, and the stored signals are read out at the same timing as that upon capture of the images in the image reproduction mode, thus removing the influence of the jitter.
However, this conventional system does not consider any case wherein the video camera 105, the pan, tilt, zoom functions and the like of which can be externally controlled, is connected to the terminal 100, and the position of the video camera 105 is controlled by a client on which an image is being displayed. This problem will be explained in detail below. In the conventional system, since the image data to be displayed are buffered by the receiving communication terminal 200, the image acquisition time has a large difference from the image display time. For this reason, when the client on which an image is being displayed controls the pan, tilt, or zoom function of the video camera 105 while observing the displayed image, a certain lag time is produced from when the user controls a function of the video camera 105 until the image corresponding to the controlled function is displayed on the client side. Hence, the user controls the video camera 105 more than he or she wanted, and e.g., the field of view of the video camera 105 moves beyond the angle that the user intended.
A problem posed when the angle (including pan and tilt angles and zoom ratio) of the camera has changed will be explained below.
For example, as shown in FIG. 4, assume that a camera which points in a direction of an object A is panned to sense the image of an object B. In this case, since the transmission data volume is constant, images are captured at a predetermined frame rate and image data is kept transmitted to the client while the camera angle is being changed from the object A to the object B.
Such images transmitted during the change from the object A to the object B may be desirable in some respects, but are not so significant for the user who knows the positional relationship between the objects A and B. In fact, such user may feel that nonsense images are transmitted/received.
When the above-mentioned camera control communication terminal (camera control device) and client are built on the network, transmission of unwanted information is preferably precluded, since it influences the operability of other users on the network.
In particular, the xe2x80x9cInternetxe2x80x9d is receiving a lot of attention. When camera control devices and clients are built on such huge network, the above-mentioned problem comes into focus inevitably.
In the system that transmits images using digital transmission media such as the Internet, in general, the sensed digital image data is compressed by a compression technique, e.g., Motion JPEG, MPEG, or the like, and the compressed image data is transmitted. The compressed image data is expanded by a device (to be referred to as a viewer hereinafter), and thereafter, the expanded image is displayed on the display.
However, in such conventional system, it is a common practice not to perform compression processing upon transmission of images, or to permanently use a single compression scheme suitable for image contents or network characteristics (transmission rate or the like), if compression processing is done. For this reason, when an uncompressed image signal is transmitted from a network with a low transmission rate or when the frame rate drops since an image is compressed by a compression scheme suitable for a high-resolution image (a scheme with a low compression rate), if the user makes camera control such as panning or tilting, images are displayed intermittently. As a result, the user fails to recognize the current state (e.g., pan or tilt angle) of the camera which is sensing an image in real time.
In such image transmission system, the images to be transmitted to all the clients are subjected to identical compression or image processing independently of whether or not the client who receives media data has the right of access to that video camera. That is, an identical media processing method is used for all the clients. For this reason, even when the client who has the right of camera access wants to observe an image with higher resolution so as to control the operation of the camera, he or she can only observe the image with a normal resolution. On the other hand, a client who has no right of camera access need not observe an image with a high resolution, and wants to display images in real time at high speed. However, such switching cannot be done in the conventional system.
On the client, image packets are stored using the image buffer (jitter buffer), and frame images to be displayed are generated using the stored packets, as described above. On the other hand, in order to satisfactorily attain camera control at the client having the right of camera access, images are preferably reproduced in real time so as to recognize images of the sensed object in real time.
The present invention has been made in consideration of the above-mentioned problems, and has as its object to provide an image sensing control method and apparatus, image transmission control method, apparatus, and system, and a storage medium that stores a program for implementing the method, which suppress the transmission volume of sensed image data until an image sensing angle, magnification of an image sensing device reaches a desired angle, so that a desired image sensing angle by an operator can be quickly obtained, and the load on the network during changes in angle and/or magnification can be reduced.
It is another object of the present invention to provide an image sensing control method and apparatus, image transmission control method, apparatus, and system, and a storage medium that stores a program for implementing the method, which change a compression method of an image signal during operation control of an image sensing apparatus, and transmit the image signal compressed by the changed compression method onto the network, so that changes in image by the operation control of the image sensing device can be detected in real time via the network.
It is another object of the present invention to provide an image sensing control method and apparatus, image transmission control method, apparatus, and system, and a storage medium that stores a program for implementing the method, which increase the compression ratio of an image signal sensed by an image sensing device when the operation of the image sensing device is controlled, and transmit the image signal compressed at a high compression rate, so that the time required for transmitting the image signal can be shortened, and any lag of image signals sensed by the image sensing device during the operation control can be reduced.
It is another object of the present invention to provide an image sensing control method and apparatus, image transmission control method, apparatus, and system, and a storage medium that stores a program for implementing the method, which can reduce any lag between the time at which the transmitter acquired images by controlling an image sensing device and the time at which a client receives and displays that image signal upon remote-control of the image sensing device, and can improve the response of displayed images with respect to the operation state of the image sensing device when the client remote-controls the image sensing device while observing the displayed images.
It is another object of the present invention to provide an image sensing control method and apparatus, image transmission control method, apparatus, and system, and a storage medium that stores a program for implementing the method, which can quickly display image signals acquired by a control device after an operation of an image sensing device when a client remote-controls the image sensing device via a control device of the image sensing device.
It is another object of the present invention to provide an image sensing control method and apparatus, image transmission control method, apparatus, and system, and a storage medium that stores a program for implementing the method, which change the processing method of an image signal to be transmitted to a client depending on whether or not the client who receives an image signal sensed by an image sensing device via a network has the right of access to the image sensing device.
It is another object of the present invention to provide an image sensing control method and apparatus, image transmission control method, apparatus, and system, and a storage medium that stores a program for implementing the method, which transmit an image signal which is converted into low-compression codes, to a client who has the right of access to an image sensing device, and transmit an image signal, which is converted into high-compression codes, to a client who has no right of access.
It is another object of the present invention to provide an image sensing control method and apparatus, image transmission control method, apparatus, and system, and a storage medium that stores a program for implementing the method, in which a client who has the right of access to an image sensing device, comprises a small-capacity jitter absorption buffer, and a client who has no right of access comprises a large-capacity jitter absorption buffer.
Other features and advantages of the present invention will be apparent from the following descriptions taken in conjunction with the accompanying drawings, in which like reference characters designate the same or similar parts throughout the figures thereof.